Skip to content

Feat/otel memory semconv v0.1.0 - #2142

Open
henrikrexed wants to merge 6 commits into
MemPalace:developfrom
henrikrexed:feat/otel-memory-semconv-v0.1.0
Open

Feat/otel memory semconv v0.1.0#2142
henrikrexed wants to merge 6 commits into
MemPalace:developfrom
henrikrexed:feat/otel-memory-semconv-v0.1.0

Conversation

@henrikrexed

Copy link
Copy Markdown

What does this PR do?

Adds opt-in OpenTelemetry instrumentation to mempalace-mcp, emitting all
three signal pillars — traces, metrics, and logs — on the core memory
operations, mapped to the memory-semconv v0.1.0 working draft.

It is off by default and zero-overhead when off: unless
OTEL_EXPORTER_OTLP_ENDPOINT is set and the [observability] extra is
installed, telemetry.py imports no SDK, allocates nothing, and every entry
point is a no-op. The default install path is unaffected.

What it emits when enabled:

  • Traces — a memory.<op> span (read / write / invalidate) around
    every MCP tool call, via a taxonomy mapping each tool to its semconv
    operation kind. Resource attributes: memory.sut.name=mempalace,
    memory.sut.architecture=mcp, service.name, service.version.
  • Metricsmemory_recall_results_count and
    memory_recall_top_similarity (histograms) recorded on search.
  • Logs — a LoggingHandler bridges MemPalace's stdlib loggers to OTLP so
    records emitted inside a span carry its trace_id/span_id for log↔trace
    correlation. Attached only to MemPalace-owned logger names, keeping the MCP
    stdio protocol clean.
  • W3C tracecontext propagation — the dispatch wrapper reads
    traceparent/tracestate from each tools/call request's _meta, so the
    memory.<op> span nests under the caller's trace: one end-to-end trace
    across the agent's MCP client and MemPalace's handling.

PII / cardinality: raw memory content is never attached to spans — only the
tool name and operation kind. Recall is captured as counts and top-similarity,
not query text.

Files:

  • mempalace/telemetry.py (new) — the whole feature behind the opt-in gate.
  • mempalace/mcp_server.py — wraps dispatch in the span + extracts inbound
    trace context (no-op when off).
  • mempalace/searcher.py — records recall metrics.
  • pyproject.toml — new optional observability extra (OTel API/SDK + OTLP
    HTTP exporter); not a core dependency.
  • docs/observability.md — enablement, env vars, backend examples, full
    span/metric/log reference.
  • docs/verification/mempalace-baseline.dql — copy-paste verification queries.
  • tests/test_telemetry.py (new) — taxonomy, no-op gate, span emission,
    context extraction, recall metrics.

License: MIT (matches the project).

How to test

# 1. Default path is unaffected (telemetry stays a no-op)
python -m pytest tests/ -v
ruff check .

# 2. Exercise the instrumentation
pip install "mempalace[observability]"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"   # any OTLP/HTTP collector
mempalace-mcp

Drive a few tool calls, then confirm memory.read / memory.write /
memory.invalidate spans — plus the two recall metrics and correlated logs —
land in your OTLP backend (Dynatrace, Grafana Tempo/Mimir, Honeycomb, Jaeger,
…). docs/verification/mempalace-baseline.dql has ready-made queries that
assert the emitted signals match the semconv contract.

image

Checklist

  • Tests pass (python -m pytest tests/ -v)
  • No hardcoded paths
  • Linter passes (ruff check .)

PerformanceChef and others added 5 commits July 27, 2026 11:45
Wires the working-draft memory-semconv v0.1.0 conventions into the
MemPalace MCP server. Off by default; activates only when both
OTEL_EXPORTER_OTLP_ENDPOINT is set AND the [observability] extra is
installed. No new runtime deps on the default install path.

Spans
  Every MCP tool dispatch emits a memory.<operation> span where
  <operation> is one of read | write | invalidate. Attributes:
  memory.operation, memory.tool. Argument values are never attached
  (PII discipline).

Metrics
  search_memories records two histograms per call:
    memory_recall_results_count   (drawers returned)
    memory_recall_top_similarity  (cosine sim of top-1 hit)

Resource attributes
  memory.sut.name=mempalace, memory.sut.architecture=mcp,
  service.name (default mempalace-mcp, OTEL_SERVICE_NAME override),
  service.version (from mempalace.version).

Files
  + mempalace/telemetry.py      (new — lazy init, no-op when disabled)
  + docs/observability.md       (new — opt-in setup + cardinality notes)
  + docs/verification/mempalace-baseline.dql (new — Dynatrace queries)
  M mempalace/mcp_server.py     (init at startup + span around dispatch)
  M mempalace/searcher.py       (record_recall after re-rank)
  M pyproject.toml              ([observability] extra; OTel API/SDK/OTLP)

Verified
  * uv sync --extra dev --extra observability — clean install
  * 193 tests pass (test_mcp_server + test_searcher + test_mcp_stdio_protection)
  * ruff check — clean
  * No-op path: env unset → init_telemetry returns False, dispatch
    yields _NOOP_SPAN, record_recall short-circuits
  * Live path: env set + collector unreachable → spans buffer, retries
    log warnings, server keeps serving (telemetry never blocks dispatch)

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the opt-in OTel integration so it now covers all three pillars
(traces, metrics, logs) and accepts trace context from MCP clients via
params._meta. An agent + MemPalace now share a single trace_id when
the client injects the W3C tracecontext headers.

Logs
  init_telemetry now also configures a LoggerProvider + OTLPLogExporter
  and bridges stdlib `logging` -> OTel via LoggingHandler attached to
  the `mempalace` logger (not root, so chromadb/posthog keep their
  existing stderr-only behavior). Every dispatch emits one structured
  record:
    "memory.dispatch tool=<tool_name> operation=<read|write|invalidate>"
  Records carry trace_id + span_id of the active span.

Trace context propagation
  New extract_trace_context(meta) parses W3C tracecontext headers
  from params._meta using TraceContextTextMapPropagator. memory_operation
  now accepts a parent_context parameter and starts the memory.<op>
  span as a child of the remote parent when supplied.

  The MCP dispatch wrapper in mcp_server.py extracts params._meta and
  passes it through, so any client that already owns a trace can fan
  into MemPalace without code changes.

  Sentinel object distinguishes "no parent" from "explicit None" so
  callers can pass either without ambiguity. Malformed headers fall
  back to a new trace and log at debug -- never raise.

Coverage
  + tests/test_telemetry.py (7 tests, no-op path)
  + 2 new DQL queries in docs/verification/mempalace-baseline.dql:
      Q7 logs with trace correlation by operation/tool
      Q8 end-to-end traces with multi-service fan-out
  M docs/observability.md -- logs section + reference client snippet
    that shows how to inject tracecontext into params._meta

Verified
  * full test suite: 190 mcp+searcher tests still pass, 7 new
    telemetry tests pass, ruff clean
  * Live tenant verification against dynatrace-dev:
    - 4 trace_ids each contain both `verify-agent` and
      `mempalace-mcp-e2e` services under one id (end-to-end
      propagation confirmed)
    - 4 `memory.dispatch tool=... operation=verify` log records
      landed in the logs pillar
  * No-op path: all new paths short-circuit when env unset

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ace loggers

Two issues caught by end-to-end testing against a real mempalace-mcp
server (the in-process smoke from the previous commit didn't surface
either):

1. The structured `memory.dispatch` log line was emitted unconditionally
   from the dispatch wrapper. On the default install path (no
   OTEL_EXPORTER_OTLP_ENDPOINT) it leaked one INFO line per call to
   stderr -- noise with no consumer. Guard with `is_enabled()` so the
   line only fires when telemetry is on and there's somewhere for it
   to go.

2. The OTel LoggingHandler was only attached to the `mempalace` logger
   (dotted). But mcp_server.py / searcher.py / palace.py / miner.py /
   fact_checker.py / query_sanitizer.py / convo_miner.py all use the
   underscore name `mempalace_mcp`, which is NOT a child of the dotted
   tree. Result: the dispatch log line landed at OTel logger setup,
   the embedding init line landed, but none of the per-call dispatch
   logs reached OTLP. Attach the handler to all five MemPalace-owned
   logger names (`mempalace`, `mempalace_mcp`, `mempalace_graph`,
   `mempalace_hallways`, `mempalace_format_miner`).

Verified
  * tests/test_mcp_server.py + tests/test_telemetry.py: 167 pass
  * full suite: 2087 passed, 3 skipped
  * ruff clean
  * /tmp/e2e_mcp_optout.py drives a real mempalace-mcp subprocess with
    no OTel env -- stderr now free of `memory.dispatch` markers; module
    state confirms no LoggingHandler attached
  * /tmp/e2e_mcp_stdio.py drives a real mempalace-mcp subprocess with
    OTel pointed at dynatrace-dev. Confirmed in DT:
      - 4 spans: memory.read(3) + memory.write(1) keyed by memory.tool
      - 4 log lines: `memory.dispatch tool=X operation=Y` per call

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The upstream fork sync (ISI-1920) added mempalace_checkpoint,
delete_by_source, delete_hallway, mine, list_hallways, and
kg_supersede. These were falling through operation_for_tool()'s
default ('read'), mislabeling four mutating tools and one fact
retraction. Classify them explicitly: checkpoint/delete_by_source/
delete_hallway/mine -> write, list_hallways -> read, kg_supersede
-> invalidate (atomic retire+write at a boundary, like kg_invalidate).

Also regenerates uv.lock for the [observability] extra against the
freshly synced dependency set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tream PR

Remove internal issue-tracker/project references (ISI-1068, ISI-1920,
"kagent") from the instrumentation docs and code comments ahead of the
upstream contribution, and format the telemetry dispatch lines to the
project's line-length=100 so `ruff format --check` passes in CI.

No behavior change: comment/whitespace only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve conflict in mempalace/mcp_server.py: nest develop's new
_write_stall_watch watchdog inside this branch's memory_operation
tracing span so both wrap the handler call (span covers the stall
watch). Adopts develop's ruff 0.16.1 pin — lint gates (ruff check /
ruff format --check) pass under 0.16.1; pin-match + telemetry +
stall-watch tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
@igorls igorls added enhancement New feature or request needs-rebase PR has merge conflicts with develop and needs rebase labels Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request needs-rebase PR has merge conflicts with develop and needs rebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants